Skip to content

ESQL: Add Connector SPI and gRPC/Arrow Flight module#142667

Open
costin wants to merge 3 commits intoelastic:mainfrom
costin:esql/connector-spi
Open

ESQL: Add Connector SPI and gRPC/Arrow Flight module#142667
costin wants to merge 3 commits intoelastic:mainfrom
costin:esql/connector-spi

Conversation

@costin
Copy link
Member

@costin costin commented Feb 18, 2026

Extend the external data source framework with a request/response
Connector SPI for services that return data over a network protocol.
Implement the first connector for Apache Arrow Flight over gRPC.

  • ConnectorFactory/Connector/ResultCursor: new SPI interfaces for
    connection-oriented external sources
  • AsyncConnectorSourceOperatorFactory: async operator that drains a
    connector on a background thread with backpressure
  • OperatorFactoryRegistry: dispatches ConnectorFactory via the
    ExternalSourceFactory hierarchy
  • FlightConnectorFactory/FlightConnector: Arrow Flight implementation
    with schema resolution and streaming result cursor
  • FlightTypeMapping: Arrow-to-ESQL type conversion
  • GrpcDataSourcePlugin: plugin wiring via ServiceLoader
  • EmployeeFlightServer: in-memory test fixture serving employee data
  • FlightFormatSpecIT: integration tests via csv-spec against a live
    Flight server
  • ExternalSourceDrainUtils: shared drain loop with timeout
  • entitlement-policy.yaml: grants manage_threads and outbound_network

Developed using AI-assisted tooling

Extend the external data source framework with a request/response
Connector SPI for services that return data over a network protocol.
Implement the first connector for Apache Arrow Flight over gRPC.

- ConnectorFactory/Connector/ResultCursor: new SPI interfaces for
  connection-oriented external sources
- AsyncConnectorSourceOperatorFactory: async operator that drains a
  connector on a background thread with backpressure
- OperatorFactoryRegistry: dispatches ConnectorFactory via the
  ExternalSourceFactory hierarchy
- FlightConnectorFactory/FlightConnector: Arrow Flight implementation
  with schema resolution and streaming result cursor
- FlightTypeMapping: Arrow-to-ESQL type conversion
- GrpcDataSourcePlugin: plugin wiring via ServiceLoader
- EmployeeFlightServer: in-memory test fixture serving employee data
- FlightFormatSpecIT: integration tests via csv-spec against a live
  Flight server
- ExternalSourceDrainUtils: shared drain loop with timeout
- entitlement-policy.yaml: grants manage_threads and outbound_network

Developed using AI-assisted tooling
@costin costin requested a review from a team as a code owner February 18, 2026 22:15
@costin costin requested a review from bpintea February 18, 2026 22:15
@elasticsearchmachine elasticsearchmachine added Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) v9.4.0 labels Feb 18, 2026
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-analytical-engine (Team:Analytics)

@elasticsearchmachine
Copy link
Collaborator

Hi @costin, I've created a changelog YAML for you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

:Analytics/ES|QL AKA ESQL >enhancement Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) v9.4.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

Comments